New in version 1.1: three new variables are provided now: lengths, maxlength and minlength
This section describes all the variables implemented in the reporting tools in the package IPCReport. When using the report.py script variables are specified with the directive --variable. However, there is a convenient shortcut to specify them as well --vvar where var shall be substituted by the corresponding variable.
There are two types of variables: raw and elaborated
Raw variables are those that read values directly from the terminal directories of the results directory —or, alternatively, from the summary that encompasses the same observation. For example, section invokeplanner.py shows all the files generated after running the sequential optimal planner bjolp with the first problem of the floortile domain. Information that can be directly retrieved by parsing the contents of these files is known as being raw.
In particular, the raw variables currently acknowledged by report.py (see report.py) are listed below:
- logfile¶
Name of the logfile generated by invokeplanner.py when running the planner on the selected problem
- vallogfile¶
Name of the logfile generated by the Automatic Validation Tool VAL when validating the results of a particular directory
- track¶
Track this case belongs to
- subtrack¶
Subtrack this case belongs to
- planner¶
Name of the planner used in this particular experiment
- domain¶
Name of the domain used in this particular experiment
- problem¶
Problem identifier (a number in the range 000 .. 999) of this particular case
- timeout¶
Timeout (in seconds) given to invokeplanner.py when making this experiment in the directive --time
- membound¶
Memory limit (in bytes) given to invokeplanner.py when making this experiment in the directive --memory
- runtime¶
Overall run time (in seconds) used by the planner for solving this particular problem. If the process started a number of threads and/or processes, this variable records the sum of all of them.
- memend¶
Memory used by the planner (in bytes) at the end of the planning process. While most planners take memory incrementally, there are some who do not do it. In this case, this is not necessarily the memory peak used by the planner
- memmax¶
Memory peak (in bytes) of the memory used by the planner during the whole planning process.
- numsols¶
Number of plan solution files generated. While in the sequential optimal track this number shall be expected to be either 0 or 1, planners shall be expected to generate an arbitrary number of plan solution files.
- valnumsols¶
Number of non-empty solution files generated. While the expected behaviour is that planners either generate plan solution files with some information or they do not, an additional check can be performed with this variable.
- oknumsols¶
Number of plan solution files that have been successfully validated by the Automatic Validation Tool VAL
- timesols¶
Time elapsed (in seconds) since the planner was started when different solutions were found. This variable can either be empty, contain a single value or an array of values.
- oktimesols¶
Time elapsed (in seconds) since the planner was started when different valid solutions (validated by the Automatic Validation Tool VAL) were found. This variable can either be empty, contain a single value or an array of values.
- timelabels¶
While the planner is running, the script invokeplanner.py samples the time and memory consumption of the planner at regular intervals. This variable records the time spent (in seconds) among successive samplings —currently, samplings are performed every 5 seconds.
- memlabels¶
Memory consumption (in bytes) at each sampling as described in timelabels
- timefirstsol¶
Time spent (in seconds) until the first solution has been found. This value is just the first one reported in timesols and might be the same than timelastsol
- timelastsol¶
Time spent (in seconds) until the last solution has been found. This value is just the last one reported in timesols and might be the same than timefirstsol
- oktimefirstsol¶
Time spent (in seconds) until the first valid solution (validated by the Automatic Validation Tool VAL) has been found. This value is just the first one reported in oktimesols and might be the same than oktimelastsol
- oktimelastsol¶
Time spent (in seconds) until the last valid solution (validated by the Automatic Validation Tool VAL) has been found. This value is just the last one reported in oktimesols and might be the same than oktimefirstsol
- solved¶
This is a boolean variable that takes the value true if the planner has generated a plan solution file and false otherwise. Whether the solution is correct or not is not taken into account in this variable.
- oksolved¶
This variable that takes the value true if the planner has generated a valid solution and false otherwise. If no solution was validated, it takes the special value ?
- plansoln¶
Filenames of non-empty solution files generated by the planner for this particular case.
- okplansoln¶
Filenames of non-empty valid solution files (validated by the Automatic Validation Tool VAL) generated by the planner for this particular case.
- values¶
Values returned by the Automatic Validation Tool VAL for each plan that was found to be valid
- uppervalue¶
Largest value returned by the Automatic Validation Tool VAL. While it is not necessarily true, this should be expected to be the first one of the values reported in values
- lowervalue¶
Lowest value returned by the Automatic Validation Tool VAL. While it is not necessarily true, this should be expected to be the last one of the values reported in values
- lengths¶
This variable is available since version 1.1. It returns a list with the number of operations in every plan validated by the Automatic Validation Tool VAL. This variable and the next two ones serve, for example, to compare the cost of the plans (which are stored in the variable values) with the number of actions in each plan.
- maxlength¶
This variable is avaliable since version 1.1. It returns the largest step length returned by the Automatic Validation Tool VAL
- minlength¶
This variable is avaliable since version 1.1. It returns the smallest step length returned by the Automatic Validation Tool VAL
A variable is said to be elaborated if it results from some computation from raw variables. In some cases, the elaborated variables might refer solely to raw variables of a particular case but most likely they result from the computation of raw variables from different cases simulatenously.
The elaborated variables currently acknowledged by report.py (see report.py) are summarized below:
- sumruntime¶
Sum of the overall runtime (raw variable runtime) of a selected subset of problems
- minruntime¶
Minimum overall runtime (raw variable runtime) of a selected subset of problems
- maxruntime¶
Maximum overall runtime (raw variable runtime) of a selected subset of problems
- summemend¶
Sum of the values of the raw variable memend of a selected subset of problems
- minmemend¶
Minimum value of the raw variable memend of a selected subset of problems
- maxmemend¶
Maximum value of the raw variable memend of a selected subset of problems
- summemmax¶
Sum of the values of the raw variable memmax of a selected subset of problems
- minmemmax¶
Minimum value of the raw variable memmax of a selected subset of problems
- maxmemmax¶
Maximum value of the raw variable memmax of a selected subset of problems
- sumnumsols¶
Total number of solution files generated for a selected subset of planning tasks
- oksumnumsols¶
Total number of solution files validated by the Automatic Validation Tool VAL
- numprobs¶
Total number of problems in a particular selection
- numsolved¶
Total number of problems for which at least one solution plan has been generated (which however might be invalid) among a selected subset of planning tasks
- oknumsolved¶
Total number of problems that have been solved with valid solutions according to the Automatic Validation Tool VAL among a selected subset of planning tasks
- numfails¶
Total number of problems not solved for a selected subset of planning tasks
- timefails¶
Cases where the planner reached the time bound without ever generating a single solution file, for a selected subset of planning tasks
- memfails¶
Cases where the planner reached the memory limit without ever generating a single solution file, for a selected subset of planning tasks
- unexfails¶
Cases where the planner failed to generate at least a solution file and it terminated without reaching the time bound or the memory limit.
- numtimefails¶
Number of timefails identified for a selected subset of planning tasks
- nummemfails¶
Number of memfails identified for a selected subset of planning tasks
- numunexfails¶
Number of unexfails identified for a selected subset of planning tasks
- minvalue¶
Minimum cost returned by the Automatic Validation Tool VAL for a selected subset of planning tasks
- maxvalue¶
Maximum cost returned by the Automatic Validation Tool VAL for a selected subset of planning tasks